{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "07_PyTorch",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "bOChJSNXtC9g",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# PyTorch"
      ]
    },
    {
      "metadata": {
        "id": "OLIxEDq6VhvZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/logo.png\" width=150>\n",
        "\n",
        "In this lesson we'll learn about PyTorch which is a machine learning library used to build dynamic neural networks. We'll learn about the basics, like creating and using Tensors, in this lesson but we'll be making models with it in the next lesson.\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/pytorch.png\" width=300>"
      ]
    },
    {
      "metadata": {
        "id": "VoMq0eFRvugb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Tensor basics"
      ]
    },
    {
      "metadata": {
        "id": "0-dXQiLlTIgz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "d4ed17af-40a8-41db-ba6e-825ff9db2187"
      },
      "cell_type": "code",
      "source": [
        "# Load PyTorch library\n",
        "!pip3 install torch"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (1.0.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "rX7Vs1JxL9wX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import torch"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Nv0xryLkKujV",
        "colab_type": "code",
        "outputId": "d46d5e58-2195-40a8-841c-26b627541a83",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "cell_type": "code",
      "source": [
        "# Creating a zero tensor\n",
        "x = torch.Tensor(3, 4)\n",
        "print(\"Type: {}\".format(x.type()))\n",
        "print(\"Size: {}\".format(x.shape))\n",
        "print(\"Values: \\n{}\".format(x))"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Type: torch.FloatTensor\n",
            "Size: torch.Size([3, 4])\n",
            "Values: \n",
            "tensor([[1.1744e-35, 0.0000e+00, 2.8026e-44, 0.0000e+00],\n",
            "        [       nan, 0.0000e+00, 1.3733e-14, 4.7429e+30],\n",
            "        [1.9431e-19, 4.7429e+30, 5.0938e-14, 0.0000e+00]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "vnyzY4PHL7c5",
        "colab_type": "code",
        "outputId": "70ed373d-e7e0-43cd-e732-51be86377721",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# Creating a random tensor\n",
        "x = torch.randn(2, 3) # normal distribution (rand(2,3) -> uniform distribution)\n",
        "print (x)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[ 0.7434, -1.0611, -0.3752],\n",
            "        [ 0.2613, -1.7051,  0.9118]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "DVwGNeKxMXI8",
        "colab_type": "code",
        "outputId": "6a185aa3-96f2-4e29-b116-3de3025cff4d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# Zero and Ones tensor\n",
        "x = torch.zeros(2, 3)\n",
        "print (x)\n",
        "x = torch.ones(2, 3)\n",
        "print (x)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[0., 0., 0.],\n",
            "        [0., 0., 0.]])\n",
            "tensor([[1., 1., 1.],\n",
            "        [1., 1., 1.]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "BPjHnDmCMXLm",
        "colab_type": "code",
        "outputId": "c14c494e-b714-4983-eb90-665064830a14",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# List → Tensor\n",
        "x = torch.Tensor([[1, 2, 3],[4, 5, 6]])\n",
        "print(\"Size: {}\".format(x.shape)) \n",
        "print(\"Values: \\n{}\".format(x))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Size: torch.Size([2, 3])\n",
            "Values: \n",
            "tensor([[1., 2., 3.],\n",
            "        [4., 5., 6.]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "mG4-CHkgMXOE",
        "colab_type": "code",
        "outputId": "2b9ed2e5-9862-480e-d0ce-d231676d7f49",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# NumPy array → Tensor\n",
        "x = torch.from_numpy(np.random.rand(2, 3))\n",
        "print(\"Size: {}\".format(x.shape)) \n",
        "print(\"Values: \\n{}\".format(x))"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Size: torch.Size([2, 3])\n",
            "Values: \n",
            "tensor([[0.0372, 0.6757, 0.9554],\n",
            "        [0.5651, 0.2336, 0.8303]], dtype=torch.float64)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "L8X2-5cqMXRA",
        "colab_type": "code",
        "outputId": "af1c82ab-b8d7-4ea6-e142-7f8ed50fda40",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# Changing tensor type\n",
        "x = torch.Tensor(3, 4)\n",
        "print(\"Type: {}\".format(x.type()))\n",
        "x = x.long()\n",
        "print(\"Type: {}\".format(x.type()))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Type: torch.FloatTensor\n",
            "Type: torch.LongTensor\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "S2BRPaMvPbe3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Tensor operations"
      ]
    },
    {
      "metadata": {
        "id": "Xrn8I76TMXT1",
        "colab_type": "code",
        "outputId": "556b9d7f-79da-415c-f85d-648c5394e3a3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# Addition\n",
        "x = torch.randn(2, 3)\n",
        "y = torch.randn(2, 3)\n",
        "z = x + y\n",
        "print(\"Size: {}\".format(z.shape)) \n",
        "print(\"Values: \\n{}\".format(z))"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Size: torch.Size([2, 3])\n",
            "Values: \n",
            "tensor([[ 0.5650, -0.0173,  1.1263],\n",
            "        [ 3.4274,  1.3610, -0.9262]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "157fC9WsMXWf",
        "colab_type": "code",
        "outputId": "a6890b43-4c74-42c6-d654-f62b8c130403",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# Dot product\n",
        "x = torch.randn(2, 3)\n",
        "y = torch.randn(3, 2)\n",
        "z = torch.mm(x, y)\n",
        "print(\"Size: {}\".format(z.shape)) \n",
        "print(\"Values: \\n{}\".format(z))"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Size: torch.Size([2, 2])\n",
            "Values: \n",
            "tensor([[ 1.3294, -2.4559],\n",
            "        [-0.4337,  4.9667]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "G6316lAmMXZG",
        "colab_type": "code",
        "outputId": "3dce79e7-1b9f-4218-84cd-afbb16af7dd4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        }
      },
      "cell_type": "code",
      "source": [
        "# Transpose\n",
        "x = torch.randn(2, 3)\n",
        "print(\"Size: {}\".format(x.shape)) \n",
        "print(\"Values: \\n{}\".format(x))\n",
        "y = torch.t(x)\n",
        "print(\"Size: {}\".format(y.shape)) \n",
        "print(\"Values: \\n{}\".format(y))"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Size: torch.Size([2, 3])\n",
            "Values: \n",
            "tensor([[ 0.0257, -0.5716, -0.9207],\n",
            "        [-1.0590,  0.2942, -0.7114]])\n",
            "Size: torch.Size([3, 2])\n",
            "Values: \n",
            "tensor([[ 0.0257, -1.0590],\n",
            "        [-0.5716,  0.2942],\n",
            "        [-0.9207, -0.7114]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "FCgDCOCjMXcF",
        "colab_type": "code",
        "outputId": "ff1e16f5-bcd9-407f-9c99-361a0b7f27f6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "cell_type": "code",
      "source": [
        "# Reshape\n",
        "z = x.view(3, 2)\n",
        "print(\"Size: {}\".format(z.shape)) \n",
        "print(\"Values: \\n{}\".format(z))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Size: torch.Size([3, 2])\n",
            "Values: \n",
            "tensor([[ 0.0257, -0.5716],\n",
            "        [-0.9207, -1.0590],\n",
            "        [ 0.2942, -0.7114]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "T3-6nGgvECH9",
        "colab_type": "code",
        "outputId": "9599adaf-1feb-4a42-d4b5-af23f1de5b2d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 561
        }
      },
      "cell_type": "code",
      "source": [
        "# Dangers of reshaping (unintended consequences)\n",
        "x = torch.tensor([\n",
        "    [[1,1,1,1], [2,2,2,2], [3,3,3,3]],\n",
        "    [[10,10,10,10], [20,20,20,20], [30,30,30,30]]\n",
        "])\n",
        "print(\"Size: {}\".format(x.shape)) \n",
        "print(\"Values: \\n{}\\n\".format(x))\n",
        "a = x.view(x.size(1), -1)\n",
        "print(\"Size: {}\".format(a.shape)) \n",
        "print(\"Values: \\n{}\\n\".format(a))\n",
        "b = x.transpose(0,1).contiguous()\n",
        "print(\"Size: {}\".format(b.shape)) \n",
        "print(\"Values: \\n{}\\n\".format(b))\n",
        "c = b.view(b.size(0), -1)\n",
        "print(\"Size: {}\".format(c.shape)) \n",
        "print(\"Values: \\n{}\".format(c))"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Size: torch.Size([2, 3, 4])\n",
            "Values: \n",
            "tensor([[[ 1,  1,  1,  1],\n",
            "         [ 2,  2,  2,  2],\n",
            "         [ 3,  3,  3,  3]],\n",
            "\n",
            "        [[10, 10, 10, 10],\n",
            "         [20, 20, 20, 20],\n",
            "         [30, 30, 30, 30]]])\n",
            "\n",
            "Size: torch.Size([3, 8])\n",
            "Values: \n",
            "tensor([[ 1,  1,  1,  1,  2,  2,  2,  2],\n",
            "        [ 3,  3,  3,  3, 10, 10, 10, 10],\n",
            "        [20, 20, 20, 20, 30, 30, 30, 30]])\n",
            "\n",
            "Size: torch.Size([3, 2, 4])\n",
            "Values: \n",
            "tensor([[[ 1,  1,  1,  1],\n",
            "         [10, 10, 10, 10]],\n",
            "\n",
            "        [[ 2,  2,  2,  2],\n",
            "         [20, 20, 20, 20]],\n",
            "\n",
            "        [[ 3,  3,  3,  3],\n",
            "         [30, 30, 30, 30]]])\n",
            "\n",
            "Size: torch.Size([3, 8])\n",
            "Values: \n",
            "tensor([[ 1,  1,  1,  1, 10, 10, 10, 10],\n",
            "        [ 2,  2,  2,  2, 20, 20, 20, 20],\n",
            "        [ 3,  3,  3,  3, 30, 30, 30, 30]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "hRtG5LShMXew",
        "colab_type": "code",
        "outputId": "b54e520a-8cd5-40a9-8b38-64919574dce0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "cell_type": "code",
      "source": [
        "# Dimensional operations\n",
        "x = torch.randn(2, 3)\n",
        "print(\"Values: \\n{}\".format(x))\n",
        "y = torch.sum(x, dim=0) # add each row's value for every column\n",
        "print(\"Values: \\n{}\".format(y))\n",
        "z = torch.sum(x, dim=1) # add each columns's value for every row\n",
        "print(\"Values: \\n{}\".format(z))"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Values: \n",
            "tensor([[ 0.4295,  0.2223,  0.1772],\n",
            "        [ 2.1602, -0.8891, -0.5011]])\n",
            "Values: \n",
            "tensor([ 2.5897, -0.6667, -0.3239])\n",
            "Values: \n",
            "tensor([0.8290, 0.7700])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "zI0ZV45PrYmw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Indexing, Splicing and Joining"
      ]
    },
    {
      "metadata": {
        "id": "iM3UFrs0MXhL",
        "colab_type": "code",
        "outputId": "bfcbbf13-d8a1-4fc1-f244-fd54068ca74b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "cell_type": "code",
      "source": [
        "x = torch.randn(3, 4)\n",
        "print(\"x: \\n{}\".format(x))\n",
        "print (\"x[:1]: \\n{}\".format(x[:1]))\n",
        "print (\"x[:1, 1:3]: \\n{}\".format(x[:1, 1:3]))"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "x: \n",
            "tensor([[-1.0305,  0.0368,  1.2809,  1.2346],\n",
            "        [-0.8837,  1.3678, -0.0971,  1.2528],\n",
            "        [ 0.3382, -1.4948, -0.7058,  1.3378]])\n",
            "x[:1]: \n",
            "tensor([[-1.0305,  0.0368,  1.2809,  1.2346]])\n",
            "x[:1, 1:3]: \n",
            "tensor([[0.0368, 1.2809]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "_tbpwGxcMXj0",
        "colab_type": "code",
        "outputId": "678e805f-f5ec-49fe-d8d6-0986a3c41672",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "cell_type": "code",
      "source": [
        "# Select with dimensional indicies\n",
        "x = torch.randn(2, 3)\n",
        "print(\"Values: \\n{}\".format(x))\n",
        "col_indices = torch.LongTensor([0, 2])\n",
        "chosen = torch.index_select(x, dim=1, index=col_indices) # values from column 0 & 2\n",
        "print(\"Values: \\n{}\".format(chosen)) \n",
        "row_indices = torch.LongTensor([0, 1])\n",
        "chosen = x[row_indices, col_indices] # values from (0, 0) & (2, 1)\n",
        "print(\"Values: \\n{}\".format(chosen)) "
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Values: \n",
            "tensor([[ 0.0720,  0.4266, -0.5351],\n",
            "        [ 0.9672,  0.3691, -0.7332]])\n",
            "Values: \n",
            "tensor([[ 0.0720, -0.5351],\n",
            "        [ 0.9672, -0.7332]])\n",
            "Values: \n",
            "tensor([ 0.0720, -0.7332])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "tMeqSQtuMXmH",
        "colab_type": "code",
        "outputId": "9fa99c82-78d9-41f8-d070-710cf1b045c7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "cell_type": "code",
      "source": [
        "# Concatenation\n",
        "x = torch.randn(2, 3)\n",
        "print(\"Values: \\n{}\".format(x))\n",
        "y = torch.cat([x, x], dim=0) # stack by rows (dim=1 to stack by columns)\n",
        "print(\"Values: \\n{}\".format(y))"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Values: \n",
            "tensor([[-0.8443,  0.9883,  2.2796],\n",
            "        [-0.0482, -0.1147, -0.5290]])\n",
            "Values: \n",
            "tensor([[-0.8443,  0.9883,  2.2796],\n",
            "        [-0.0482, -0.1147, -0.5290],\n",
            "        [-0.8443,  0.9883,  2.2796],\n",
            "        [-0.0482, -0.1147, -0.5290]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "JqiDuIC-ByvO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Gradients"
      ]
    },
    {
      "metadata": {
        "id": "qxpGB7-VL7fs",
        "colab_type": "code",
        "outputId": "a7964762-60d4-4e0e-bed2-b2d392804494",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "cell_type": "code",
      "source": [
        "# Tensors with gradient bookkeeping\n",
        "x = torch.rand(3, 4, requires_grad=True)\n",
        "y = 3*x + 2\n",
        "z = y.mean()\n",
        "z.backward() # z has to be scalar\n",
        "print(\"Values: \\n{}\".format(x))\n",
        "print(\"x.grad: \\n\", x.grad)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Values: \n",
            "tensor([[0.7014, 0.2477, 0.5928, 0.5314],\n",
            "        [0.2832, 0.0825, 0.5684, 0.3090],\n",
            "        [0.1591, 0.0049, 0.0439, 0.7602]], requires_grad=True)\n",
            "x.grad: \n",
            " tensor([[0.2500, 0.2500, 0.2500, 0.2500],\n",
            "        [0.2500, 0.2500, 0.2500, 0.2500],\n",
            "        [0.2500, 0.2500, 0.2500, 0.2500]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "uf7htaAMDcRV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* $ y = 3x + 2 $\n",
        "* $ z = \\sum{y}/N $\n",
        "* $ \\frac{\\partial(z)}{\\partial(x)} = \\frac{\\partial(z)}{\\partial(y)} \\frac{\\partial(y)}{\\partial(x)} = \\frac{1}{N} * 3 = \\frac{1}{12} * 3 = 0.25 $"
      ]
    },
    {
      "metadata": {
        "id": "VQtpZh1YD-kz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# CUDA tensors"
      ]
    },
    {
      "metadata": {
        "id": "E_C3en05L7iT",
        "colab_type": "code",
        "outputId": "01b0eddc-db28-4786-ae48-a1004c838186",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Is CUDA available?\n",
        "print (torch.cuda.is_available())"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "True\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "za47KWEJ6en2",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "If the code above return False, then go to `Runtime` → `Change runtime type` and select `GPU` under `Hardware accelerator`. "
      ]
    },
    {
      "metadata": {
        "id": "BY2DdN3j6ZxO",
        "colab_type": "code",
        "outputId": "ec0ac0bd-461d-4b45-e131-cbf1d19c955b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Creating a zero tensor\n",
        "x = torch.Tensor(3, 4).to(\"cpu\")\n",
        "print(\"Type: {}\".format(x.type()))"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Type: torch.FloatTensor\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "EcmdTggzEFPi",
        "colab_type": "code",
        "outputId": "0e3326db-8d3d-40aa-accd-b31ab841b572",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Creating a zero tensor\n",
        "x = torch.Tensor(3, 4).to(\"cuda\")\n",
        "print(\"Type: {}\".format(x.type()))"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Type: torch.cuda.FloatTensor\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}
